Image Processing Device, Image Processing Method 
and Image Processing Program 
for Joining a Plurality of Divided Images 

5 This application is based on an application No. 2000-26624 filed in 

Japan, the content of which is hereby incorporated by reference. 
BACKGROUND OF THE INVENTION 
Field of the Invention 

The present invention relates to an image processing device, an 
10 image processing method and an image processing program. 

Particularly, the invention relates to an image processing device, an 
image processing method and an image processing program for producing 
an image of a high resolution by joining a plurality of images obtained 
from divided portions of a subject. 
15 Description of the Related Art 

Such a technique has been known that can obtain an image of high 
resolution by joining a plurality of partial images obtained from divided 
portions of a subject. This is an effective technique for obtaining high- 
resolution images in the case where a resolution of an acquirable image is 
20 restricted due to the number of pixels of a charge coupled device (which 
will be referred to as a "CCD" hereinafter) or the like. This technique is 
disclosed, e.g., in Japanese Patent Laying-Open Nos. 6-164890, 5-12430 
and 6-98101. 

In the joining techniques disclosed therein, characteristic points 
25 and a correlation in change of pixel values are detected, and positioning 
or registration is performed based on the relative positional relationship 
between the images thus obtained. 

In the conventional joining technique, however, regions of the 
partial images to be joined may not accurately overlap with each other 
30 due to shift in position occurred at the time of taking partial images by 
the CCD or the like, distortions in lens, affine strain or the like. In this 
case, unnatural images or blurred images are formed in a joint or border 
between joined images, resulting in low image quality. 
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Such a problem also arises that long times are required for 
detecting corresponding characteristic points in the divided images to be 
joined and detecting the correlation in change of the pixel values. 
SUMMARY OF THE INVENTION 
5 The invention has been developed for overcoming the above 

problems, and an object of the invention is to provide an image processing 
device, an image processing method, which can prevent deterioration of a 
quality of an image formed of a plurality of images joined together, and 
an image processing program achieving such prevention. 

10 Another object of the invention is to provide an image processing 

device, an image processing method, in which a speed of processing of 
joining a plurality of images is increased, and an image processing 
program achieving such increase in speed. 

According to an aspect, the invention for achieving the above 

15 objects provides an image processing device for producing an entire image 
of a subject by joining a plurality of divided images produced from divided 
portions defined in the subject and having partially overlapping portions, 
including a setting portion for setting a plurality of sets each including 
corresponding points in the two divided images having overlap regions 

20 overlapped together, a transforming portion for performing geometric 

transformation of one or the other of the two divided images based on the 
plurality of corresponding point sets, and a joining portion for joining the 
two divided images based on the plurality of corresponding point sets 
after the geometric transformation, wherein higher importance is placed 

25 on a specific portion of the overlapping regions compared with the other 
in the geometric transformation. 

According to another aspect, the invention provides an image 
processing device for joining at least first and second divided images 
produced from divided portions defined in a subject and having partially 

30 overlapping portions, including a detecting portion for detecting a relative 
positional relationship between the first and second divided images 
having overlap regions overlapped together, a setting portion for setting a 
plurality of sets each including corresponding points in the two divided 
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images within the overlapping regions based on the positional 
relationship detected by the detecting portion, and a joining portion for 
joining the two divided images based on the set corresponding point sets. 
According to still another aspect, the invention provides an image 
5 processing method including the steps of obtaining a plurality of divided 
images produced from divided portions defined in a subject and having 
partially overlapping portions, and producing an entire image 
representing the subject by joining the plurality of produced divided 
images, wherein the joining of the divided images is performed while 
10 placing higher importance on a specific portion among the overlapping 
regions. 

According to still another aspect, the invention provides an image 
processing method including the steps of obtaining a plurality of divided 
images produced from divided portions defined in a subject and having 

15 partially overlapping portions, detecting a direction of positional shift 

between the two divided images having the overlap regions overlapping 
with each other, setting a plurality of sets each including corresponding 
points in the two divided images based on the detected positional shift 
direction, and joining the two divided images based on the set 

20 corresponding point sets. 

According to further another aspect, the invention provides an 
image processing program causing a computer to execute the steps of 
obtaining a plurality of divided images produced from divided portions 
defined in a subject and having partially overlapping portions, and 

25 producing an entire image representing the subject by joining the 

plurality of produced divided images, wherein the joining of the divided 
images is performed while placing higher importance on a specific portion 
among the overlapping regions. 

According to further another aspect, the invention provides an 

30 image processing program causing a computer to execute the steps of 
obtaining a plurality of divided images produced from divided portions 
defined in a subject and having partially overlapping portions, detecting 
a direction of positional shift between the two divided images having the 
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overlap regions overlapping with each other, setting a plurality of sets 
each including corresponding points in the two divided images based on 
the detected positional shift direction, and joining the two divided images 
based on the set corresponding point sets. 
5 According to the invention, it is possible to prevent blurring and/or 

other conspicuous states of joined portions of the images. Therefore, the 
invention can provide the image processing device and the image 
processing method, which can prevent deterioration of a quality of the 
image completed by joining the plurality of images, and can also provide 
10 the image processing program. 

The foregoing and other objects, features, aspects and advantages 
of the present invention will become more apparent from the following 
detailed description of the present invention when taken in conjunction 
with the accompanying drawings. 
15 BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a block diagram showing a circuit structure of a scanner of 
an embodiment of the invention; 

Fig. 2 shows an example of an original document; 

Figs. 3A - 3D show four partial images obtained from four divided 
20 portions of the original shown in Fig. 1; 

Fig. 4 shows a first method of detecting corresponding points; 

Fig. 5 shows a second method of detecting the corresponding 
points; 

Fig. 6 shows a result of joining the four partial images obtained 
25 from an original divided into four; 

Fig. 7 is a view showing processing of detecting corresponding 
points corresponding together in upper left and upper right partial 
images; 

Fig. 8 is another view showing processing of detecting 
30 corresponding points corresponding together in upper left and upper right 
partial images; 

Fig. 9 shows a result of joining nine partial images obtained from 
an original divided into four; 
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Figs. 10A - IOC show joining processing for joining two partial 
images by butt joint; 

Figs. 11A - 11C show a principle of joining processing using 
weighted average; 

5 Figs. 12A and 12B fragmentarily show by way of example an upper 

left partial image and an upper right partial image to be joined; 

Figs. 13A and 13B show a result of joining with different degrees of 
importance placed on corresponding points; 

Fig. 14 is a flowchart showing a flow of joining processing 
10 performed by a CPU of a scanner of the embodiment; 

Fig. 15 is a flowchart showing flows of processing of detecting a 
correlation between partial images in a step S04 shown in Fig. 14; and 

Figs. 16 - 19 are flowcharts showing flows of correlation detecting 
processing performed in steps Sll - S14 shown in Fig. 15, respectively. 
15 DESCRIPTION OF THE PREFERRED EMBODIMENTS 

A scanner provided with an image processing device of an 
embodiment of the invention will now be described with reference to the 
drawings. In the figures, the same or corresponding portions bear the 
same reference numbers, and description thereof is not repeated. 
20 Fig. 1 is a block diagram showing a circuit structure of a scanner in 

an embodiment of the invention. Referring to Fig. 1, a scanner 1 
includes a central processing unit (which will be referred to as a "CPU" 
hereinafter) for performing entire control of scanner 1, a CCD 104 serving 
as an image-taking element for taking or obtaining an image, a moving 
25 portion 106 for moving CCD 104 and thereby changing relative positions 
of CCD 104 and a subject, a random access memory 108 (which will be 
referred to as an "RAM" hereinafter) for temporarily storing an image 
sent from CCD 104, a correlation detecting portion 110 for detecting a 
correlation between the plurality of images stored in RAM 108, a 
30 corresponding point detecting portion 112 for detecting a plurality of 

corresponding points, which are present in two images to be joined and 
are in the corresponding relationship, a transformation parameter 
detecting portion 114 for detecting transformation parameters to be used 
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for geometric transformation of the image, a joining portion 116 for 
joining two images together based on the detected transformation 
parameters and the corresponding points, an output portion 118 for 
outputting the joined images, and a read only memory 102 (which will be 
5 referred to as an "ROM" hereinafter) for storing a program to be executed 
by CPU 100. 

An external storage device 122 may be connected to CPU 100 of 
scanner 1 for reading a program for controlling scanner 1 from a CD- 
ROM 124, magneto-optical disk, digital video disk, floppy disk or the like. 

10 In this case, the program for executing joining processing to be described 
later by CPU 100 is stored in the record medium such as CD-ROM 124, 
and external storage device 122 reads the program, whereby CPU 100 
can execute the program. In the case where CPU 100 executes the 
program for performing the joining processing, it is not necessary to 

15 provide correlation detecting portion 110, corresponding point detecting 
portion 112, transformation parameter detecting portion 114 and joining 
portion 116. 

The joining processing which will be described later may be 
performed within scanner 1. Data may be set to another scanner or a 

20 computer terminal connected to scanner 1 for performing the processing 
therein. In this case, the foregoing joining processing program is sent 
into the personal computer from the record medium such as a CD-ROM, 
and is executed by the CPU of the personal computer. The personal 
computer is provided with an interface for capturing divided images 

25 obtained by the scanner connected thereto. 

Scanner 1 of this embodiment obtains partial images from divided 
portions of an original, and joins the plurality of divided partial images to 
output one image. Specifically, this embodiment can be applied, for 
example, to a digital camera or a scanner such as a scanner BC3000 

30 manufactured by Minolta Co., Ltd., which can take an images of, e.g., a 
page(s) of a book from a higher side. For simplicity reason, scanner 1 
will now be described in connection with an example, in which images are 
obtained from four portions prepared by dividing an original into four, 
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and the obtained four partial images are joined together. 

The original may be divided into a number other than four, and 
may be divided into a number larger than four or an arbitrary even or 
odd number. 

5 Referring to Fig. 1, CCD 104 starts integration in accordance with 

an instruction sent from CPU 100, and dumps the CCD data to RAM 108 
after the above integration is completed. Thereby, RAM 108 stores one 
partial image. 

Moving portion 106 moves the image-taking position of CCD 104 in 

10 accordance with the instruction sent from CPU 100 so that CCD 104 can 
take partial images from the plurality of different portions. For 
obtaining the images by scanner 1 from the divided four portions, the 
original is divided into upper left, upper right, lower left and lower right 
portions. Moving portion 106 moves the position of CCD 104 so that one 

15 of the divided four portions of the original may be located within an 
image-taking range of CCD 104. The positions of CCD 104 is 
determined such that the neighboring two partial images overlap with 
each other. Scanner 1 may be configured to move the original instead of 
moving CCD 104. Neither CCD 104 nor the original may be moved, and 

20 an optical system between them may be moved to change the image- 
taking region for obtaining divided images. 

The operations of moving CCD 104 by moving portion 106 and 
taking an image by CCD 104 are repeated to obtain upper left, upper 
right, lower left and lower right images from the upper left, upper right, 

25 lower left and lower right portions of the original, respectively. 

Fig. 2 shows an example of the original. Figs. 3A - 3D show an 
example of the four partial images obtained by dividing the original 
shown in Fig. 2 into four. Fig. 3A shows an upper left partial image P, 
Fig. 3B shows an upper right partial image Q, Fig. 3C shows a lower left 

30 partial image R and Fig. 3D shows a lower right partial image S. 

An overlap region 130A at the right of upper left partial image P 
overlaps with an overlap portion 130B at the left of upper right partial 
image Q. An overlap portion 132A at the bottom of upper left partial 
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image P overlaps with an overlap portion 132B at the top of lower left 
partial image R. Likewise, an overlap portion 131B at the left of lower 
right partial image S overlaps with an overlap portion 131A at the right 
of lower left partial image R. An overlap portion 133B at the top of 
5 lower right partial image S overlaps with an overlap portion 133A at the 
bottom of upper right partial image Q. 

These overlapped portions 130A and 130B contain the 
substantially same partial image of the subject except for an error in 
images. Likewise, overlapped portions 131A and 131B, 132A and 132B, 

10 and 133A and 133B contain the substantially same partial images of the 
subject except for errors in images. 

Correlation detecting portion 110 of scanner 1 detects correlation 
between the plurality of portions stored in RAM 108. The correlation 
represents the relative positional relationship between the two partial 

15 images to be joined. It is preferable that CCD 104 obtains the partial 

images in such a state that the partial images adjoining in the horizontal 
direction match with each other in the vertical direction, and the partial 
images adjoining in the vertical direction match with each other in the 
horizontal direction. However, shifts in the vertical or horizontal 

20 directions may often occur in actual cases. For example, left and right 
upper partial images P and Q, which are shown in Figs. 2A and 2B, 
respectively, shift vertically from each other, and upper and lower left 
partial images P and R shift horizontally from each other. 

The relative positional relationship is expressed by one between 

25 the two partial images to be joined, and particularly the partial image 

shifted toward the center of the original. For example, the relationship 
between left and right upper partial images P and Q shown in Figs. 3A 
mad 3B is represented by the partial image shifted downward because 
the center of the original is shifted downward with respect to them. The 

30 relationship between left and right lower partial images R and S is 

represented by the partial image shifted upward because the center of the 
original is shifted upward. The relationship between upper and lower 
left partial images P and R is represented by the partial image shifted 
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rightward because the center of the original is shifted rightward. The 
relationship between upper and lower right partial images Q and S is 
represented as the partial images shifted leftward because the center of 
the original is shifted leftward. 
5 The relative positional relationship can be easily detected in the 

following manner. The plurality of partial images stored in RAM 108 
are reduced into an image of a lower resolution, the two partial images 
selected from the reduced partial images are roughly positioned using a 
correlation between pixel values at and around the vertical or horizontal 
10 centers of the selected two partial images. Alternatively, comparison is 
made between distributions in horizontal or vertical projective 
histograms of the two partial images. Thereby, the partial image shifted 
toward the center of the original can be easily determined between the 
partial images. 

15 The purpose of using the reduced image of a low resolution is to 

improve the processing speed. Therefore, it is not necessary to increase 
a processing speed, it is not necessary to use the reduced images. 

Based on the output of moving portion 106, RAM 108 may store the 
positions (i.e., upper left, upper right, lower left and lower right positions) 

20 of the respective partial images together with the partial images obtained 
by CCD 104. This facilitates determination of the relative positional 
relationship. 

Corresponding point detecting portion 112 of scanner 1 detects the 
corresponding points, which are common to the partial images to be 

25 joined, based on the relative positional relationship detected by 

correlation detecting portion 110. The corresponding points obtained by 
corresponding point detecting portion 112 are stored in RAM 108. 

The two partial images can be joined in such a manner that the 
two partial images are overlapped together in all the possible 

30 combinations for achieving the highest correlation, and thereby the 

correlation between the images is obtained. In this case, the correlation 
is first obtained from the reduced images, and then the resolution is 
gradually increased. Difference values between the two images or the 
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like are used for expressing the correlation. According to this method, 
the correlation can be obtained with certain accuracy and within a short 
processing time if the two images can be joined by dealing with only 
parallel shift without requiring geometric transformation. If the 
5 geometric transformation, e.g., for rotary movement or change in scale 

factor is required for joining the two images, it is preferable for improving 
the processing speed to determine the correlation locally or region by 
region. 

There are two methods for detecting the corresponding points 

10 corresponding to each other in the two partial images. Fig. 4 shows a 
first method of detecting the corresponding points. In the first method 
shown in Fig. 4, characteristic points 141A - 141D included in one (141) of 
partial images 141 and 142 and characteristic points 142A - 142D 
included in the other partial image 142 are detected, and each of 

15 characteristic points 141A - 141D included in partial image 141 is 

compared with all characteristic points 142A - 142D included in other 
partial image 142 for determining whether the correlation is present or 
not. According to this method, each of the characteristic points included 
in one of the partial images is compared with all the characteristic points 

20 included in the other partial image for determining the presence of the 
corresponding relationship. Therefore, this method requires a long 
processing time. 

Fig. 5 shows the second method of detecting the corresponding 
points. In the second method shown in Fig. 5, characteristic points 143A 

25 - 143D included in one (143) of the partial images are detected, and the 

characteristic points corresponding to detected characteristic points 143A 
- 143D are detected from other partial image 144. 

The characteristic point can be, e.g., an edge, a point of a 
maximum pixel value, a point of a minimum pixel point obtained by 

30 secondary differential filtering. The characteristic points are not 

restricted to them obtained by the above manner, and may be obtained by 
other known methods. The characteristic point may not be a point, but 
may be a characteristic region formed of a set of points. 
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The correlation between the characteristic points can be detected 
by obtaining a sum of absolute values of the differences between the 
images of the block having a center defined by the characteristic points 
included in one of the partial image and the block having a center defined 
5 by the characteristic points included in the other image, and determining 
the combination of the blocks providing the minimum sum of the absolute 
values. 

The second method has such an advantage that the required 
processing time is shorter than that of the first method. However, in the 

10 case where one partial image 143 is shifted downward with respect to 
other partial image 144 as shown in Fig. 5, a characteristic point 
corresponding to characteristic point 143D included in one partial image 
143 is not present in other partial image. In this case, such a problem 
may occur that an incorrect characteristic point which is not originally in 

15 a corresponding relationship is erroneously detected in other partial 
image 144, or an extra processing time is required for detecting the 
corresponding point. 

This problem can be overcome by detecting the corresponding 
characteristic points based on the partial image located in a further outer 

20 position. The further outer position is determined from the relative 

positions of the two partial images to be joined, and is determined as a 
position remote from a region where three or more images including the 
other partial image overlap with each other. The partial image in the 
further outer position is detected based on the output of correlation 

25 detecting portion 111. 

Fig. 6 shows a result of four partial images obtained from the 
original divided into four. Four partial images P, Q, R and S are joined 
together so that four partial images overlap with each other in an overlap 
region 151. Overlap region 151 is located substantially in a center of an 

30 image 150 formed of four partial images P, Q 5 R and S joined together. 

Two partial images P and Q in the upper side are positioned such 
that upper right partial image Q is shifted away from overlap portion 151 
with respect to left partial image P. Therefore, upper right partial 
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image Q is in the partial image, which is located in the further outer 
position, and is the partial image forming the reference. 

Upper left partial image P and lower left partial image R are 
relatively positioned such that upper left partial image P is shifted away 
5 from overlap portion 151 with respect to lower left partial image R. 

Therefore, upper left partial image P is in the partial image, which is 
located in the further outer position, and is the partial image forming the 
reference. Likewise, lower left partial image R and lower right partial 
image S are relatively positioned such that lower left partial image R is 
10 in the further outer position, and form the reference. Upper and lower 
right partial images Q and S are relatively positioned such that upper 
right partial image Q is in the further outer position, and forms the 
reference. 

Among four partial images P, Q, R and S shown in Fig. 6, the 

15 correspondence between the characteristic points, which are included in 
upper two partial images P and Q, will now be discussed. Upper right 
partial image Q forming the reference has an upper end, which does not 
overlap with upper left partial image P. It is clear that upper left 
partial image P does not contain the characteristic point corresponding to 

20 the characteristic point which is present in the above region which does 
not overlap. Therefore, characteristic points, which are included in 
upper right partial image Q but are remote from overlap portion 151, are 
not detected for performing appropriate processing. 

Fig. 7 shows processing for detecting corresponding points in 

25 partial images P and Q. Upper right partial image Q forming the 

reference is equally divided into four, and characteristic points 152A and 
152B included in the lower three regions among the four regions thus 
divided are detected. Characteristic points 153A and 153B which 
correspond to characteristic points 152A and 152B, respectively, are 

30 detected in upper left partial image P. If the plurality of characteristic 
points are detected in each region, the leftmost character point is 
employed. This is because the probability that the leftmost 
characteristic point is included in the left partial image P is the highest. 
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This increases the probability that the characteristic point corresponding 
to the characteristic point detected from the upper right partial image Q 
forming the reference is detected from the upper left partial image P. 
Therefore, an extra processing time is not required. 
5 In contrast to this, such a manner may be employed that upper left 

partial image P which is not in the further outer position is used as the 
reference, the characteristic points are detected from the upper three 
regions among the four regions defined by vertically dividing upper left 
partial image P into four, and the characteristic points corresponding to 

10 the detected characteristic points are detected in upper right partial 

image Q. The characteristic points included in the lowermost region of 
upper left partial image P are located at or around the center of image 
150 formed of the joined partial images. Since changes in quality of the 
portion at and near the center of image 150 are more conspicuous that 

15 those at and near the end of image 150. Therefore, the corresponding 

points at and near the center are more important than those at and near 
the end. Accordingly, it is preferable to detect the corresponding points 
using the upper right partial image Q in the further outer position as a 
reference. 

20 The partial image is equally divided into four. If the amount of 

the relative position shift between the two partial images to be joined can 
be detected by correlation detecting portion 110, such a manner may be 
employed that the characteristic points included in the region 
corresponding to this shift amount are not detected. 

25 There is also a manner in which the partial image handled as the 

reference is changed depending on the positions of the corresponding 
points. Fig. 8 is another view showing detection of the corresponding 
points in partial images P and Q. When the corresponding points are 
detected in the upper halves of partial images P and Q shown in Fig. 8, 

30 upper left partial image P in the further inner position is used as the 
reference. When the corresponding points are detected in the lower 
halves, upper right partial image Q in the further outer position is used 
as the reference. The partial image in the further inner position is the 
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partial image shifted toward overlap region 151 with respect to the other, 
and the partial image in the further outer position is the partial image 
shifted away from overlap region 151 with respect to the other. 

In addition to the relative positional relationship between the two 
5 partial images to be joined, the positions of the characteristic points in 
the partial image can be used as the reference for changing the partial 
image forming the reference, whereby the corresponding points can be 
detected more reliably. 

Fig. 9 shows a result of joining nine partial images obtained from 

10 the original divided into nine. The detection of the corresponding points 
from four or more partial images to be joined is performed using the 
partial image, which is shifted away from the overlapping region of the 
three partial images, as a reference. For example, if three partial 
images 160, 161 and 163 are joined together, a region 169 where three 

15 partial images 160, 161 and 163 overlap with each other, is present. In 
the relative positional relationship between partial images 160 and 161, 
partial image 160 is shifted outward from overlap region 169 to a larger 
extent. Therefore, the detection of the corresponding points in the 
partial images 160 and 161 is performed using partial image 160 as a 

20 reference. Likewise, in the relative positional relationship between 
partial images 160 and 163, partial image 160 is shifted outward from 
overlap region 169 to a larger extent. Therefore, the detection of the 
corresponding points in the partial images 160 and 163 is performed 
using partial image 160 as a reference. 

25 The foregoing manner of detecting the corresponding points can be 

used not only for the case of joining the three or more partial images but 
also for the case of joining the two partial images. For example, if 
determination of the relative positional relationship (i.e., determination 
of the partial image in the upper position) is not be performed in 

30 connection with the left and right divided partial images, the range for 
detection of the characteristic points, which are used for the 
corresponding point detection, can be defined in advance within the 
reference partial image, whereby the corresponding points can be 
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detected. 

However, the range for detecting the characteristic points must be 
located at or around the vertical center of the left partial image. 
Therefore, the detected corresponding points are concentrated at and 
5 around the center of the image. Therefore, upper or lower end of the 
joined images may blur. Further, the characteristic points may not be 
detected if the characteristic region is present in the upper or lower end, 
and is not present in the central portion. 

In contrast to the above, the method of detecting the corresponding 

10 points already described with reference to Figs. 7 and 8 can be used if the 
relative positional relationship between the left and right partial images 
is already known. For example, when it is already known that the left 
partial image is shifted downward with respect to the right partial image, 
the lower portion of the right partial image is reliably present within the 

15 left partial image if an extreme difference in size is not present between 
the two partial images. Therefore, the characteristic points of the lower 
region of the right partial image can be detected using the right partial 
image as a reference, and the characteristic points corresponding to the 
detected characteristic points can be reliably detected from the left 

20 partial image. In this manner, the optimum corresponding points can be 
reliably detected by using the relative positional relationship between the 
two partial images even for joining the two partial images. 

Parameter detecting portion 114 of scanner 1 obtains 
transformation parameters for performing geometric transformation of 

25 the partial images based on the corresponding points obtained by 

corresponding point detecting portion 112. Affine transformation or 
Helmert's transformation may be used for transformation of the partial 
images. 

Description will now be given on the calculation of transformation 
30 parameters for joining the left and right partial images. The 

transformation parameters can be determined based on F, which is the 
transformation parameter of the right partial image using the left partial 
image as the reference, and more specifically, can be determined by 
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obtaining F, which minimize a value S in the following evaluation 
formula. 

S = I(Pi-F(Qi))2 (1) 
where Pi represents the position of the characteristic point in the left 
5 partial image, and Qi represents the position of the characteristic point in 
the right partial image. 

Further, the following evaluation formula (2), which is prepared by 
giving high weight to the foregoing evaluation formula (1), can be used 
for placing importance on a specific characteristic point pair (Pi, Qi). 
10 Sk = ZKi(Pi - F(Qi))2 • • • (2) 

where Ki shows weight given to the respective characteristic points. 

In the foregoing formula (2), a coefficient Ki can be increased for 
obtaining transformation parameter F for specific points on which 
importance is placed. 
15 If the degree of importance on the specific point can be changed 

depending on the position in the overlapping region, the methods of 
calculating transformation parameter F and method of placing 
importance on the characteristic point are not restricted to the foregoing 
method. 

20 In the process of joining partial images P, Q, R and S obtained by 

dividing the original into four as shown in Fig. 6, the quality at and 
around the center of completed image 150 is improved by increasing the 
importance on the corresponding points included in overlap region 151 or 
the neighboring region. Likewise, in the process of joining the nine 

25 partial images prepared by dividing the original or subject into nine as 
shown in Fig. 9, the quality of completed image 150 can be improved by 
increasing the degree of importance of the corresponding points, which 
are included in the overlap region (e.g., overlap region 169) defined by at 
least the three partial images or the neighboring region. 

30 The degree of importance may be determined depending on the 

number of the partial images overlapping in the overlap region. The 
degree of importance is increased in the region, where more partial 
images overlap with each other, or the neighboring region. Thereby, it is 
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possible to prevent reduction in quality of completed image 150. 

Joining portion 116 of scanner 1 joins the partial images stored in 
RAM 108 based on the corresponding points detected by corresponding 
point detecting portion 112 and the transformation parameters detected 
5 by transformation parameter detecting portion 114. The joining method 
may be roughly divided into a method of butt joint and a method using a 
weighted average. 

In the joining process utilizing the butt joint, two partial images 
are joined along certain line. In the joining process utilizing the 

10 weighted average, the pixel values of the pixels within the overlapping 
regions of the two partial images to be joined are averaged with variable 
weights depending on the positions, and the average value thus obtained 
is used as the pixel value in the overlap region for performing the joining. 
The joining process utilizing the butt joint can be effectively used 

15 for regions, where a change in pixel value between a certain pixel and a 
neighboring pixel is small, as can be seen, e.g., in a character region 
containing characters. However, the joining process utilizing the butt 
joint is unpreferable for a region, where pixel values continuously change, 
as can be seen in a photograph region containing a photograph or the like, 

20 because joint or border line becomes conspicuous. 

Figs. 10A, 10B and 10C show joining of the two partial images in 
the joining process utilizing the butt joint. Figs. 10A and 10B show 
partial images to be joined, and Fig. 10C shows a result of the joining 
process performed by butting the two partial images on certain line. The 

25 partial image shown in Fig. 10A and the partial image shown in Fig. 10B 
are joined together on line 155. Line 155 extends across a graph 
contained in both the two partial image. 

In contrast to this, the joining process utilizing the weighted 
average is performed in such a manner that the average value is obtained 

30 by varying the weights depending on the position in the overlap region 
between the two partial images, and is used as the pixel value in the 
overlap region. Therefore, this process is effective for the joining in the 
region such as a photograph region, where the pixel value continuously 
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varies. 

Figs. HA, 11B and 11C show a principle of the joining process 
utilizing the weighted average. The example shown in Figs. 11 relates 
to joining of the left and right partial images. Fig. 11A shows change in 
5 pixel value, which occurs in the horizontal direction within the left 

partial image. Fig. 11B shows change in pixel value, which occurs in the 
horizontal direction within the right portion. The pixel values shown in 
Figs. 11A and 11B correspond to each other. In Fig. 11A, pixel value 
change 170 is defined by the pixel values in horizontal pixel positions 
10 within the left partial image. In Fig. 11B, pixel value change 171 is 

defined by the pixel values in horizontal pixel positions within the right 
partial image. 

Fig. 11C shows overlapping using the weighted average. 
Referring to Fig. 11C, pixel value changes 170 and 171 in the left and 

15 right partial images are combined to provide pixel change 172, which is 
represented by solid line 172 in Fig. 11C, and are defined by the average 
values obtained with variable weights depending on the pixel position in 
the overlap region. According to the joining process using the weighted 
average, as described above, the weighted averages in the left and right 

20 partial images are used in the overlap region between the left and right 
partial images so that a joint or border between the joined images is not 
conspicuous. 

Accordingly, the joining process using the weighted average is 
effective for both the character region and the photograph region. 

25 Joining portion 116 of scanner 1 of the embodiment can perform either 

the joining process utilizing the butt joint or the joining process using the 
weighted average. Description will now be given on an example of the 
joining process using the weighted average, which can be used for either 
the character region or the photograph region. 

30 Joining portion 116 performs geometric transformation such as 

affine transformation on the partial images to be joined, based on 
transformation parameters obtained by transformation parameter 
detecting portion 114. The two partial images are joined together based 



- 18- 



on the corresponding points obtained by corresponding point detecting 
portion 112. 

The transformation parameters, which are detected by 
transformation parameter detecting portion 114 based on the 
5 corresponding points obtained by corresponding point detecting portion 

112, ensures the matching between the two partial images in and around 
the position where the corresponding points of the two images to be 
joined are present. However, the two partial images obtained by CCD 
104 may not completely match with each other in a portion of the overlap 

10 region, where the two partial images overlap with each other, and 
particularly in the portion other than the portion in and around the 
position where the corresponding points are present. 

If the detection of the corresponding points is performed using fast 
detection algorithm, the correspondence in a local region is detected with 

15 a precision of about one pixel unit. Therefore, the transformation 

parameter, which is detected based on the corresponding point detected 
with this precision, may contain an error smaller than one pixel. 

As described above, If the partial images to be joined do not match 
with each other in the region, where the partial images overlap with each 

20 other, but are joined with the weighted average, the completed image 

may be blurred or doubled so that the image quality lowers. This will be 
described below in greater detail, 

Figs. 12A and 12B show an example of the partial images to be 
joined. Fig. 12A shows a portion of the left partial image, and Fig. 12B 

25 shows a portion of the right partial image. The images shown in Figs. 
12A and 12B are included in the partial image portions in the overlap 
region, respectively. The partial images shown in Figs. 12A and 12B do 
not completely match with each other. It is now assumed that four 
corresponding points 180, 181, 182 and 183 are detected in Figs. 12A and 

30 12B. 

Figs. 13A and 13B show results of joining performed with different 
degrees of importance placed on the corresponding points. Figs. 13A 
shows the result of joining with the weighted average, which is 
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determined with higher importance placed on corresponding points 180 
and 181. Figs. 13B shows the result of joining with the weighted 
average, which is determined with higher importance placed on 
corresponding points 180 and 183. In Fig. 13A, the corresponding points 
5 on which the importance is placed are shifted upward. The images may 
be subjected to the geometric transformation using the transformation 
parameter, which is obtained with the importance placed on 
corresponding points 180 and 181 shifted upward, and the images thus 
subjected to the geometric transformation may be joined by the 

10 processing using the weighted average. In this case, an error in 

geometric transformation occurs in the lower portion. Therefore, the 
lower portions of the joined images are blurred or doubled. 

In contrast to this, as shown in Fig. 13B, the images may be 
subjected to the geometric transformation using the transformation 

15 parameter, which is obtained with the importance placed on vertically 
spaced points 180 and 181, and the images thus subjected to the 
geometric transformation may be joined by the process using the 
weighted average. In this case, an error in geometric transformation 
can be suppressed, and the deterioration in quality of the joined images 

20 can be prevented. 

Accordingly, the degree of importance of the corresponding points, 
which are used for detecting the transformation parameters from the 
plurality of corresponding points detected by corresponding point 
detecting portion 112, as well as the degree of importance of the 

25 corresponding points used in the joining process are determined based on 
the following bases. 

(1) The corresponding points to be selected increase the degree of 
importance of the corresponding points which are spaced from each other 
as far as possible. In other words, it is determined to increase the 

30 degree of importance of the corresponding points such that the selected 
corresponding point and the corresponding point are spaced by a 
predetermined distance. 

(2) It is determined to increase the degree of importance of the 
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corresponding points included in and around the region, where the 
plurality of images overlap with each other. 

The item (2) may also be expressed as "the degree of importance of 
the corresponding point near the center of the image formed of the joined 
5 images is increased". 

In this invention, "to place importance on the corresponding point 11 
and "not to place importance on the corresponding point" include concepts 
of "to utilize the corresponding point" and "not to utilize the 
corresponding point" in addition to the concepts of "to increase the 

10 importance on the corresponding point" and "not to increase the 
importance on the corresponding point", respectively. 

If the corresponding points deemed to be important are distributed 
or dispersed, it is possible to prevent deterioration of the quality of the 
joined images, as compared with the case where the corresponding points 

15 deemed to be important are not distributed. The distributed 

corresponding points may be detected in such a manner that the partial 
image is forcedly divided into a plurality of portions, and the 
characteristic points detected in the respective divided portions are 
utilized for detecting the corresponding points. Also, the corresponding 

20 points may be detected from the plurality of obtained corresponding 
points by utilizing the corresponding points in or near the upper and 
lower ends when the horizontally shifted partial images are to be joined, 
or by utilizing the corresponding points in or near the left and right ends 
when the vertically shifted partial images are to be joined. 

25 In this embodiment, weighting coefficient Ki in the evaluation 

formula (2) is used for "placing the importance". However, the 
importance may be placed based on the distribution of the corresponding 
points used for evaluation. Thus, the evaluation may be performed in 
accordance with the evaluation formula (1) while employing more 

30 corresponding points in and around the region, where the plurality of 
divided images overlap, as well as the region near the center of the 
completed image formed of the joined partial images. 

In the case where the two partial images obtained from the 
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original divided into two are to be joined together, the joining may be 
performed while placing the importance on the corresponding points near 
the center of the completed image, whereby the high quality can be 
achieved in the region which is located in a middle portion of the image, 
5 and thus is conspicuous. 

The embodiment has been described in connection with the method 
of joining the two partial images. However, it may be envisaged to 
employ such a method that four partial images obtained from the original 
divided into four are simultaneously joined together to complete one 

10 image plane. In this case, the importance of the corresponding points is 
determined based on the correspondence between the two partial images 
to be joined, and therefore the positions of the corresponding points as 
well as the relative positional relationship between the partial images 
can be determined and handled similarly to the case of completing the 

15 image by joining the two partial images. 

In scanner 1 of the embodiment, as described above, the geometric 
transformation and joining of the partial images are performed with the 
high importance placed on the corresponding points in and around the 
overlap region, where the four partial images obtained from the original 

20 divided into four overlap with each other. Therefore, it is possible to 
prevent lowering of the quality in the region where the plurality of 
partial images overlap. Further, the region where three or more partial 
images overlap is located near the center of the completed image. 
Therefore, it is possible to prevent lowering of the quality in the central 

25 region of the image. 

Among the corresponding points used for joining the two partial 
images, the importance is placed on the corresponding points in the 
distributed positions, respectively. Therefore, it is possible to prevent 
such a disadvantage that the completed image is blurred or doubled due 

30 to inappropriate geometric transformation in the joining process. 

Further, the corresponding points are detected based on the 
relative positional relationship between the partial images to be joined. 
Therefore, the corresponding points can be detected reliably. It is not 
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necessary to perform extra comparison and detection for detecting the 
corresponding points so that the processing speed can be improved. 

The corresponding points are detected based on the relative 
positions of the partial images to be joined as well as the positions of the 
5 corresponding points in the respective partial images. Therefore, the 
corresponding points can be detected more reliably. 

Description will now be given on the flow of the joining processing 
performed by CPU 100 in scanner 1. In this case, it is not necessary to 
use correlation detecting portion 110 ? corresponding point detecting 
10 portion 112, transformation parameter detecting portion 114 and joining 
portion 116. The joining program is stored in a CD-ROM 124, and an 
external storage portion 122 reads out the program stored in CD-ROM 
124 for executing it by CPU 100. 

Fig. 14 shows a flowchart of the joining processing performed by 
15 CPU 100 in scanner 1 of the embodiment. Fig. 14 shows the processing 
for joining the four partial images obtained from a subject which is 
divided vertically and horizontally into four. The joining processing 
performed by CPU 100 is not restricted to the case of handling the four 
partial images, and can be applied to all the cases handling two or more 
20 partial images. 

For the joining process shown in Fig. 14, image-taking by CCD 104 
and movement of CCD 104 by moving portion 106 are repeated. 
Thereby, partial images are obtained from the subject divided into four, 
and the partial images thus obtained are stored in RAM 108 (step S01). 
25 The four partial images stored in RAM 108 have overlap regions each 
overlapping with the neighboring image(s). 

In step S01, CPU 100 sends control signals to CCD 104 and 
moving portion 106 so that the partial images obtained by CCD 104 are 
stored in RAM 108. 

30 In a next step S02, the four partial images stored in RAM 108 are 

reduced into four reduced images, respectively. A correlation between 
the four reduced images is detected (step S03). The correlation between 
the reduced images is the relative positional relationship between the 
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neighboring two images. For example, the relative positional 
relationship between the upper left partial image P and the upper right 
partial image Q can be detected by determining, in each of the reduced 
images prepared by reducing the respective partial images, a distribution 
5 of a horizontal projective histogram of pixel values in the overlap region 
of the reduced images, and comparing the determined distributions thus 
determined. Thereby, it is possible to determine whether each of the two 
partial images is located at a higher or lower position with respect to the 
other. The positional relationship between the upper and lower left 

10 partial images P and R can be detected by obtaining and comparing the 
distributions of the projective histograms of the reduced images prepared 
by reducing the respective partial images. Thereby, it is possible to 
determine whether each of the upper and lower left partial images P and 
Q is located on the right or left of the other partial image. 

15 In a step S04, the correlation between the partial images is 

detected (step S04). For the correlation between the partial images, the 
corresponding points in the neighboring partial images are detected. 
The processing in step S04 will be described later. 

In a next step S05, transformation parameters between the partial 

20 images are calculated. Calculation of the transformation parameters is 
similar to the foregoing processing performed in transformation 
parameter detecting portion 114, and therefore description thereof is not 
repeated. 

Based on the corresponding points obtained in step S04 and the 
25 transformation parameters obtained in step SOS, upper left partial image 
P and upper right partial image Q are joined together (step S06). 

In a step S07, the two lower partial images, i.e., lower left partial 
image R and lower right partial image S are joined together. The 
images joined together in step S06 are joined to the images joined 
30 together in step S07 (step S08). The joining processing performed in 

steps S06, S07 and S08 are similar to those performed in joining portion 
116 described before, and therefore description thereof is not repeated. 

Fig. 15 is a flowchart showing a flow of the processing, which is 



-24- 



performed in step S04 in Fig. 14 for detecting the correlation between the 
partial images. Referring to Fig. 15, the correlation between the upper 
left partial image P and upper right partial image Q are detected in a 
step Sll. In a step S12, the correlation is detected between the lower 
5 left partial image R and lower right partial image S. In a step S13, the 
correlation between upper and lower left partial images P and R is 
detected. In a step S14, the correlation between the upper and lower 
right partial images Q and S is detected. 

Fig. 16 is a flowchart showing a flow of the processing, which is 

10 performed in step Sll in Fig. 15 for detecting the correlation. From the 
correlation between the reduced images obtained in step S03 shown in 
Fig. 14, it is determined in a step S21 shown in Fig, 16 whether upper 
left partial image P is shifted upward from upper right partial image Q or 
not. This is performed for the determination of the relative positions 

15 between the upper left and upper right partial images. If the upper left 
partial image is shifted upward with respect to the upper right partial 
image, the process moves to a step S22. Otherwise, the process moves to 
a step S25. 

In step S22, upper left partial image P is used as a reference image, 
20 and this reference image is vertically divided into four in a step S23. In 
a step S24, the following processing is effected on the lower three regions 
among the four divided regions prepared in step S23. 

(1) Characteristic points are detected. 

(2) The characteristic point in the rightmost position is selected 
25 from the detected characteristic points. 

In a step S25, upper right partial image Q is set as the reference 
image, and this reference image is vertically divided into four in a step 
S26. 

Among the divided four regions, the following processing is effected 
30 on the lower three regions. 

(1) Characteristic points are detected. 

(2) The characteristic point in the left end is selected from the 
detected characteristic points. 
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Thereby, one characteristic point in the leftmost position is 
selected in each of the lower three regions. 

In a step S28, the characteristic point corresponding to the 
characteristic point in the reference image is detected from the other 
5 partial image (i.e., upper right partial image Q if the reference image is 
upper left partial image P, and upper left partial image P if the reference 
image is upper right partial image Q). The characteristic points thus 
detected are handled as the corresponding points. 

Fig. 17 is a flowchart showing a flow of correlation detecting 
10 processing performed in step S12 shown in Fig. 15. Referring to Fig. 17, 
it is determined in step S31 whether lower left partial image R is shifted 
downward with respect to lower right partial image S or not. If shifted, 
the process moves to a step S32. If not, the process moves to a step S35. 

In a step S32, lower left partial image R is set as the reference 
15 image, and this reference image is vertically divided into four in a step 

S33. Among the divided four regions, the following processing is effected 
on the upper three regions in a step S34. 

(1) Characteristic points are detected. 

(2) The characteristic point in the rightmost position is selected 
20 from the detected characteristic points. 

In a step S35, lower right partial image S is set as the reference 
image, and this reference image is vertically divided into four in a step 
S36. Among the divided four regions, the following processing is effected 
on the upper three regions in a step S37. 
25 (1) Characteristic points are detected. 

(2) The characteristic point in the leftmost position is selected from 
the detected characteristic points. 

In a step S38, the characteristic point corresponding to the 
characteristic point in the reference image is detected from the other 
30 partial image (i.e., lower right partial image S if the reference image is 
lower left partial image R, and lower left partial image R if the reference 
image is upper right partial image Q). 

In this manner, the three corresponding points are detected in each 
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of the lower left and lower right partial images. 

Fig. 18 is a flowchart showing a flow of the correlation detecting 
processing performed in step S13 shown in Fig. 15. Referring to Fig. 18, 
it is determined in step S41 whether upper left partial image P is shifted 
5 leftward with respect to lower left partial image R or not. If shifted, the 
process moves to a step S42. If not, the process moves to a step S45. 

In step S42, upper left partial image P is set as the reference image, 
and this reference image is vertically divided into four in a step S43. 
Among the divided four regions, the following processing is effected on 
10 the three right regions in step S44. 

(1) Characteristic points are detected. 

(2) The characteristic point in the lowermost position is selected 
from the detected characteristic points. 

In a step S45, lower left partial image R is set as the reference 
15 image, and this reference image is vertically divided into four in a step 

S46. Among the divided four regions, the following processing is effected 
on the three right regions in a step S47. 

(1) Characteristic points are detected. 

(2) The characteristic point in the uppermost position is selected 
20 from the detected characteristic points. 

In a step S48, the characteristic point corresponding to the 
characteristic point in the reference image is detected from the other 
partial image (i.e., lower left partial image R if the reference image is 
upper left partial image P, and upper left partial image P if the reference 
25 image is lower left partial image R). 

In this manner, the three corresponding points are detected in each 
of the upper and lower left partial images P and R. 

Fig. 19 is a flowchart showing a flow of the correlation detecting 
processing performed in step S14 shown in Fig. 15. Referring to Fig. 19, 
30 it is determined in step S51 whether upper right partial image Q is 

shifted rightward from lower right partial image S or not. If shifted, the 
process moves to a step S52. If not, the process moves to a step S55. 

In a step S52, upper right partial image Q is set as the reference 
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image, and this reference image is vertically divided into four in a step 
S53. Among the divided four regions, the following processing is effected 
on the three left regions in step S54. 

(1) Characteristic points are detected. 
5 (2) The characteristic point in the lowermost position is selected 

from the detected characteristic points. 

In a step S55, lower right partial image S is set as the reference 
image, and this reference image is vertically divided into four in a step 
S56. Among the divided four regions, the following processing is effected 
10 on the three left regions in a step S57. 

(1) Characteristic points are detected. 

(2) The characteristic point in the uppermost position is selected 
from the detected characteristic points. 

In a step S58, the characteristic point corresponding to the 

15 characteristic point in the reference image is detected from the other 
partial image (i.e., lower right partial image S if the reference image is 
upper right partial image Q, and upper right partial image Q if the 
reference image is lower right partial image S). 

In this manner, the three corresponding points are detected in each 

20 of the upper and lower right partial images Q and S. 

According to the joining process performed by CPU 100 in scanner 
1 of this embodiment, as described above, the corresponding points 
corresponding together and located in the two partial images to be joined 
are detected from each of the three regions among the divided four 

25 regions. Therefore, the corresponding points spaced by a predetermined 
distance are detected. Therefore, the partial images, which are 
subjected to the geometric transformation based on the transformation 
parameters calculated in step SOS, are transformed in the optimum 
manner so that it is possible to prevent lowering of the quality of the 

30 image completed by joining. 

Since the corresponding points, which are present in the partial 
images to be joined and correspond to each other, can be reliably detected, 
the processing time can be reduced. 
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Further, the corresponding points are detected by dividing the 
reference image into four. Therefore, there is a high possibility that the 
corresponding point is detected in the region, where the partial image 
overlaps with the overlap region defined by the overlapping portions of 
5 the other two partial images to be joined. Therefore, the transformation 
parameter can be obtained while placing importance on the 
corresponding points in the region near the center of the image formed of 
the joined images. As a result, it is possible to prevent deterioration of 
the quality in the central portion of the image formed of the joined 
10 images. 

Although the present invention has been described and illustrated 
in detail, it is clearly understood that the same is by way of illustration 
and example only and is not to be taken by way of limitation, the spirit 
and scope of the present invention being limited only by the terms of the 
15 appended claims. 
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